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SUMMARY 


This  paper  describes  the  methodology  in  which  the  Tools  for 
Automated  Knowledge  Engineering  (TAKE)  software  application  has  been 
used  to  conduct  various  military  human-system  interface  evaluations. 
TAKE  is  a  prototype  computer  software  package  developed  and  designed 
by  Armstrong  Laboratories  to  elicit,  record  and  analyze  information  used 
in  system  evaluations. 

The  TAKE  methodology  involves  collecting  relevant  information 
through  a  knowledge  elicitation  technique  known  as  "Concept  Mapping." 
Concept  Mapping  is  more  "user-oriented"  than  other  techniques  such  as 
written  questionnaires  or  oral  interviews.  The  information  collected 
through  Concept  Mapping  is  more  useful  because  its  shared  medium  of 
communication  ensures  an  accurate  portrayal  of  the  system,  as  viewed  by 
the  user,  unbiased  by  any  preconceived  notions  of  the  questionnaire 
author  or  interviewer. 

TAKE  consists  of  several  knowledge  engineering  tools.  These  include 
the  capability  to  draw,  copy,  edit,  save  and  print  concept  maps,  to  produce 
hierarchical  outlines  of  concept  maps  and  to  use  customized  subject 
categories  to  search  one  or  more  concept  maps  for  user-specified 
information.  Armstrong  Laboratory  has  used  TAKE  within  the  context  of 
the  described  methodology  for  the  evaluation  and/or  design/redesign  of 
various  military,  human-system  interfaces.  Applications  for  TAKE  can  also 
include  the  evaluation  of  industrial  workplaces,  artificial  intelligence 
systems  and  other  human-system  interfaces. 
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INTRODUCTION 


This  paper  describes  the  Tools  for  Automated  Knowledge  Engineering 
(TAKE)  methodology  used  by  certain  personnel  at  Armstrong  Laboratory 
for  evaluation  and  design/redesign  of  various  military,  human-system 
interfaces.  This  methodology  can,  however,  be  applied  to  the  evaluation 
and  design/redesign  of  any  type  of  system.  The  main  difference  between 
the  TAKE  methodology  and  other  previous  methodologies  lies  in  the  type 
of  knowledge  elicitation  techniques  and  computer  analysis  tools  that  are 
utilized. 


Typical  Knowledge  Acquisition  in  System  Evaluation 

The  first  step  in  the  evaluation  of  any  system  is  acquiring  the 
information  about  a  system  that  will  enable  one  to  effectively  perform  the 
evaluation.  The  sources  of  such  information  include  reference  documents, 
system  diagrams  and  domain  experts  (DE's)  (experienced  system  users). 
Eliciting  information  from  domain  experts  typically  involves  asking  system 
related  questions  in  a  personal  interview  or  through  a  written 
questionnaire.  The  questions  asked  of  the  domain  expert  are  usually 
derived  from  the  viewpoint  of  the  system  evaluator  and  as  such,  are 
biased  in  that  direction. 

Previous  research  has  shown  that  certain  constraints  can  be  placed 
on  knowledge  acquisition  by  the  types  of  probes  made  by  the  knowledge 
engineer  (Perfetto,  Bransford,  &  Franks,  1983;  Adams,  Kasserman, 
Yearwood,  Perfetto,  Bransford,  &  Franks,  1988).  The  information  acquired 
can  be  limited  by  preconceived,  closed  end  questions  asked  of  the  domain 
expert.  Knowledge  acquisition  methods  produce  more  information  when 
they  use  open  ended  questions  (if  questions  are  used  at  all)  that  are  not 
biased  by  the  elicitor's  preconceived  ideas  about  the  system.  These 
questions  are  "product-oriented"  and  not  "product-user-oriented."  (See 
Zaff,  McNeese  and  Snyder,  1991,  for  a  more  detailed  discussion). 

In  contrast,  user-oriented  or  "user-centered"  knowledge  acquisition 
focuses  on  the  system  user  when  developing  the  knowledge  acquisition 
strategy.  This  allows  the  user  to  provide  system  information  in  a  manner 
which  is  most  comfortable  for  the  user.  Supplying  the  user  with  an  open- 
ended,  free  flowing  format  for  providing  his/her  expert  knowledge 
increases  the  chances  of  the  user  providing  the  most  relevant  and/or 
important  information  for  the  system  evaluation.  The  expert  knowledge 
acquired  is  viewed  as  important  by  the  user  and,  as  such,  is  usually 
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extremely  relevant  to  the  system  evaluation.  An  open-ended  format  also 
provides  the  user  with  the  opportunity  to  offer  feedback  to  the 
interviewer  on  the  accuracy  of  the  interviewer's  interpretation  of  what  the 
user  is  saying.  This  is  not  possible  when  completing  a  verbal  or  written 
questionnaire.  Therefore,  user-centered  knowledge  acquisition  could  be 
defined  as:  1)  involving  a  methodology  that  facilitates  flexible  access  of 
information,  2)  providing  a  medium  for  shared  communications,  and  3) 
involving  procedures  which  are  compatible  with  the  way  that  the  domain 
expert  thinks  about  his/her  system. 

A  knowledge  acquisition  technique  which  satisfies  the  definition  of 
user-centered  knowledge  acquisition  is  "Concept  Mapping."  The  TAKE 
system  evaluation  process  utilizes  concept  mapping  to  capture  information 
collected  from  domain  experts.  Concept  mapping  proved  to  be  a  superb 
knowledge  acquisition  technique  for  design  during  the  development  of  the 
Advanced  Knowledge  and  Design  Acquisition  Methodology  (AKADAM) 
(McNeese,  Zaff,  Peio,  Snyder,  Duncan  &  McFarren,  1990).  TAKE  was  created 
from  the  ideas  generated  and  the  knowledge  gained  while  utilizing 
AKADAM.  TAKE  applies  concept  mapping  to  create  integrated  knowledge 
databases  that  are  then  analyzed  with  computer  assistance  to  distill  and 
extract  information  that  allows  the  human  factors  engineer  to  develop 
sound  human-machine  interfaces. 


Concept  Mapping 

Concept  mapping  is  a  technique  where  textual  or  verbal  information 
is  collected  and  graphically  represented  in  the  form  of  interconnected 
concept  elements.  These  concept  elements  are  connected  by  links  which 
represent  the  relationships  between  the  different  concepts  (concept- 
relationship-concept).  See  Figure  1  for  a  graphic  depiction  of  a  concept- 
relationship-concept  "triple."  The  concepts  are  typically  represented  by 
ovals  (nodes)  and  the  relationships  by  uni-directional  arrows  (links). 

Concept  mapping  is  intended  as  a  means  of  collecting  and  visually 
representing  different  pieces  of  knowledge  or  information  about  a 
particular  topic  and  how  those  pieces  of  information  are  interconnected. 
Concept  mapping  is  often  used  as  a  learning  tool,  to  graphically  portray 
both  general  and  specific  concept  knowledge.  The  technique  is  easy  to  use 
and  avoids  strict  rule  sets  associated  with  more  "structured"  analysis 
techniques  (Snyder,  McNeese  and  Zaff,  1991).  See  Figure  2  below  for  an 
example  of  a  small  concept  map  of  some  of  the  advantages  of  concept 
mapping. 
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Concept 


Relationship 
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,  f  Objects 

Nodes  =  Concepts  j  Actions 

1  Events 

Arrows  =  Relationships 


Figure  1.  Example  of  a  Concept-Relationship-Concept  "triple." 


Figure  2.  Example  of  a  concept  map. 


User-centered  knowledge  acquisition  involves  a  methodology  that 
facilitates  flexible  access  of  information.  Concept  mapping  fulfills  this 
requirement.  When  a  domain  expert  is  "mapped,  they  are  able  to  free 
associate  on  a  particular  system,  it's  characteristics,  performance 
attributes,  shortcomings,  strengths  and/or  idiosyncrasies.  No  matter  what 
the  system  attribute  is,  it  and  all  of  its  inter-relations  can  be  accurately 
represented  in  a  concept  map. 

User-oriented  knowledge  acquisition  also  provides  a  medium  for 
shared  communications.  A  shared  medium  for  communications  allows  the 
conveyor  of  the  information  to  see  what  the  recipient  of  the  information 
has  comprehended  and  how  it  is  interpreted.  A  shared  medium  of 
communication  is  not  provided  for  in  the  usual  personal  interview  of 
domain  experts.  This  is  because  it  does  not  allow  all  parties  to  know  and 
track  the  interpretations  of  the  knowledge  engineer.  The  visual 
representation  of  expert  knowledge  through  interconnecting  concepts 
linked  by  relationships,  provides  an  excellent  medium  for  shared 
communications.  In  the  concept  map,  the  domain  expert  can  literally  see 
the  knowledge  engineer's  perception  of  the  system  concepts  and  their 
respective  inter-relationships  and  can  therefore  make  corrections  to  the 
map,  if  necessary. 

Additionally,  user-centered  knowledge  acquisition  requires  that  the 
knowledge  acquisition  procedures  be  compatible  with  the  way  the  domain 
expert  thinks  about  his/her  system.  This  is  accomplished  by  using  both 
structured  (e.g.  questionnaire  type)  and  unstructured  (e.g.  concept  maps) 
techniques.  This  has  the  benefit  of  eliciting  a  broad  and  deep  range  of 
information  from  the  domain  expert  while  also  satisfying  the  structured 
information  requirements  of  the  system  engineers,  programmers  and 
designers. 


Tools  for  Automated  Knowledge  Engineering  (TAKE) 

TAKE  is  a  prototype  computer  software  package  developed  at 
Armstrong  Laboratories  which  is  designed  to  analyze  information 
contained  in  concept  maps.  In  addition  to  providing  concept  map  drawing 
and  storage  capabilities,  TAKE  possesses  several  knowledge  engineering 
tools.  These  include  the  capability  to  a)  build  outlines  of  concept  maps,  b) 
facilitate  creation  of  user-defined  categories  of  key  words  and  c)  search 
one  or  more  concept  maps  for  the  key-word-related  information. 

The  drawing  capability  enables  the  user  to  enter  and  store  a  graphics 
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file  of  a  concept  map.  The  concepts  can  be  graphically  represented  by 
several  different  shapes  (ovals  are  the  default  shape).  These  are  typically 
referred  to  as  "nodes."  The  relationships  between  the  concepts  are 

graphically  represented  by  arrows  extending  from  one  concept  to  another. 
The  arrows  are  referred  to  as  "links."  Concept  and  link  labels  can  be 

easily  assigned  or  edited  simply  by  clicking  onto  the  label  with  a  mouse  or 
trackball  device  and  then  inputting  the  label  name  into  a  field  via 
keyboard  entries.  These  labels  serve  to  describe  the  concepts  and  the 
relationship  between  them. 

Concepts  and  links  can  be  created  or  deleted  with  single  keyboard 
input  commands.  Multiple  concepts  can  be  linked  together.  The  position 
of  one  or  more  concepts  within  a  map  can  be  changed  by  clicking  onto  the 
concept(s)  and  dragging  it  (them)  to  the  desired  position.  The  default  map 
size  is  one  standard  size  piece  of  paper  (8.5  x  11  in.)  but  can  be  increased 
to  up  to  approximately  100  (10  x  10)  pages. 

TAKE  is  capable  of  organizing  the  concept  map  information  into  a 

data  base,  from  which  outlines  can  be  created  and  keyword  searches  made. 
An  outline  consists  of  a  listing  of  all  the  concept-relationship-concept 
triples  in  a  top-down  hierarchical  order.  This  order  emulates  the  top- 

down  order  of  the  concept  map.  An  example  of  an  outline  is  shown  in 
Figure  3.  The  outlines  can  be  used  to  support  HSI  analyses  by  grouping 
information  into  logical  clusters.  This  feature  is  useful  in  generating 
functional  decompositions  and  in  organizing  large  scale  data  bases.  An 
ASCII  file  of  the  outline  can  be  exported  for  use  in  a  word  processor  or  for 
further  data  analyses. 

Source  Nod*  Unk  D— tint  too  Nod* 


Task  1  raq  Controls 
Controls  Inctuda  COU 
Controls  Include  Throttle 

Task  1  nq  Displays 
Displays  Ineludo  CDU 
Displays  Inctudo  FUR 
Task  1  has  Workload 
Workload  equal  to  High 
Task  1  has  Frequency 
Fraquancy  of  2 
Task  1  has  Criticality 
Criticality  aqua!  to  Medium 

Figure  3.  Example  of  a  concept  map  outline. 
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TAKE  can  also  be  used  to  index  the  concept  map  with  "context  slices." 
Context  slices  are  categories  of  information  identified  by  a  set  of  key 
words.  The  key  words  can  be  used  to  search  for  specific  types  of 
information  related  to  the  goals  of  the  analysis.  Key  words  are  assigned, 
by  the  analyst,  to  each  of  several  categories.  The  information  within  the 
concept  map  can  be  easily  categorized  for  problem  identification,  task 
assignments  or  other  characterization. 


Categories  such  as  "visual,"  "temporal,"  "spatial,  resources, 
"requirements"  and  "differences"  have  been  used  in  previous  analyses. 
Key  words  assigned  to  the  "visual"  category  included  anything  that  dealt 
with  the  human  processing  of  visual  information,  e.g.  "see,"  "look,"  "NVG," 
"glare."  The  map  data  base  is  searched  for  the  core  components  of  each 
key  word  and  any  concept-relationship-concept  triple  containing  these 
components  is  listed.  The  results  of  the  keyword  search  can  be  printed 
and/or  stored  in  a  file  under  a  filename  of  the  user  s  choice.  Later  the  file 
can  be  accessed  simply  by  clicking  on  to  the  category  name  in  the  category 
create/edit  window.  An  infinite  number  of  categories  of  information  can 
be  created  and/or  combined. 


The  categories  of  information  can  also  be  color  coded.  This  allows 
one  to  quickly  locate  specific  topic  areas  in  the  map.  Combinations  of 
categories  may  be  created  which  can  aid  in  the  determination  of  very 
specific  problem  areas  such  as  "visual  limitations  or  anthropometric 
limitations."  Figure  4  contains  the  "Categories"  window  in  which  the 
keyword  search  operations  are  performed. 
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Current  Categories 
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HI  Regular 

m 

Visual 
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5 

HU  Combination 

T 

Cannot  see  behind 

Node 

Unable  to  move  or  view 

Node 

u 

Enter/  Modify  Category 
Category  Name:  I  Limitations 
Type:  ®  Regular  O  Combination 

Keywords: 


not,  unable,  cant 


Duplicate)  Add 


Color: 


Figure  4.  TAKE  "Categories"  create/edit  window. 
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METHODOLOGY  DESCRIPTION 


The  TAKE  methodology  is  graphically  depicted  in  Figure  5.  There  are 
eight  basic  groups  of  activity  which  are  somewhat,  but  not  exclusively, 
sequential  in  nature.  These  steps  are  also  not  all  strictly  physical 
activities.  Some  steps  involve  cognitive  analyses  that  are  ongoing,  yet 
transparent  to  the  outside  observer. 


Identification  of  System/Mission  Objectives 

The  system/  mission  objectives  are  typically  provided  by  the  group 
in  need  of  the  system  evaluation  in  the  form  of  text  reference  material  or 
through  a  personal  interview.  These  objectives,  however,  can  also  be 
acquired  by  concept  mapping  various  senior  system/mission  experts.  The 
process  of  acquiring  the  system/mission  objectives  is  graphically  depicted 
in  Figure  6. 


Identification  of  System  Operations/Mission  Profile 

The  process  of  identifying  the  system  operations  or  mission  profile 
involves  four  activities  which  are  shown  in  Figure  7  and  described  below. 

1.  Identification  of  Senior  Domain  Expert  -Identification  of  a  Senior 
Domain  Expert(s)  (SDE)  is  performed  through  consultation  with  the  point  of 
contact  or  high  ranking  official  of  the  organization  requesting  the  human- 
system  interface  evaluation/redesign.  The  senior  domain  expert(s)  is 
typically  that  person  possessing  the  most  experience  operating  the  system 
being  evaluated.  They  should  be  intimately  familiar  with  all  scenarios 
within  which  the  system  operates  and  the  system  performance 
requirements  in  those  scenarios. 

2.  System  Operation/Mission  Profile  Acquisition  -  A  description  of 
the  system  operation  or  in  the  case  of  a  military  aircraft,  the  mission 
profile,  is  obtained  through  an  in-depth  interview  with  the  senior  domain 
expert(s).  The  SDE  is  asked  to  describe  the  system  operating  scenarios  and 
the  mission  to  be  accomplished.  The  information  is  recorded  on  audio  tape 
and  is  drawn  on  a  large  white/black  board  in  concept  map  form.  This 
provides  immediate  feedback  to  the  SDE  on  the  interviewer's 
interpretation  of  the  information  provided  by  the  SDE.  The  concept  map  is 
entered  into  a  computer  via  the  TAKE  software  for  later  analysis  and  hard 
copy  printout. 
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Figure  5.  TAKE  System  Evaluation/Redesign  Methodology. 


3.  Reference  Material  Acquisition  -  Reference  materials  describing 
the  system  operations  or  mission  profile  are  collected.  More  specifically, 
documents  such  as  the  User's  Manuals,  Statement  of  Operational  Need 
(SON),  Concept  of  Operations  (CONOPS),  Technical  Orders  (TOs)  and  mission 
requirements  lists  are  collected.  This  is  done  through  comprehensive, 
multi-database  literature  searches  and  solicitation  of  SDEs  for  applicable 
reference  materials. 

4.  Concept  Map  &  Reference  Material  Assimilation  -  System 
evaluation/redesign  personnel  study  and  assimilate  collected  reference 
materials  and  the  system  operation/mission  profile  concept  map  collected 
from  the  SDE. 


System  Hardware  Examination 

The  system  being  evaluated  is  examined  and  its  physical  dimensions 
and  characteristics  are  assimilated.  The  following  are  three  activities 
which  are  performed  collectively  or  individually  to  develop  a  cognitive  or 
mental  map  of  the  system  hardware  and  its  human-system  interface  (HSI). 
This  process  is  graphically  summarized  in  Figure  8. 

1.  System  Reference  Material  Acquisition  -  Reference  materials 
describing  the  physical  make-up  of  the  system  are  acquired  through 
comprehensive,  multi-database  literature  searches.  Documents  may 
include  User's/Reference  Manuals,  Technical  Orders  and  updates,  mission 
checklists,  human  factors  and  engineering  research  reports  on  various 
system  related  topics  and  diagrams  and/or  drawings  of  the  system  and  its 
HSI. 


2.  System  Reference  Material  Examination  -  Using  the  reference 
materials  the  system  evaluation  personnel  familiarize  themselves  with  the 
system's  controls,  displays,  anthropometric  characteristics,  overall  layout, 
system  personnel  staffing  and  assigned  and  unassigned  tasks. 

3.  Personal  or  Vicarious  System  Interaction  -  System  evaluation 
personnel  should  always  attempt  to  gain  some  experience  operating  the 
targeted  system  or  a  relatively  high  fidelity  system  simulator.  If  this  is 
not  possible,  then  they  should  observe  and/or  videotape  trained  system 
users  operating  the  system  in  typical  system  operating  scenarios. 
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Figure  8.  System  Hardware  Examination. 


Questionnaire  Development 


A  questionnaire  is  developed  to  elicit  information  which  may  be  of 
value  in  improving  the  HSI  but  which  may  not  be  volunteered  by  the 
DE(s).  This  information  is  often  associated  with  areas  in  which  the  DE  is 
not  even  aware  of  an  existing  problem.  For  example,  a  DE  may  not  know 
that  replacement  of  the  digital  readouts  for  several  system  states  with 
analog  representations  will  reduce  his/her  workload.  Therefore,  a  DE  may 
not  volunteer  this  information.  However,  a  properly  designed 
questionnaire  will  reveal  the  fact  that  the  HSI  in  question  is  limited  to 
digital  readouts  only,  thus  clearing  the  way  for  future  HSI  improvements 
in  this  area.  This  process  is  described  below  and  shown  in  Figure  9. 

1.  Acquisition  of  Similar  System  Evaluation  -  Obtain  (through 
literature  searches  and  personal  contact  with  field-specific  professionals) 
references  reporting  evaluations  of  systems  similar  to  the  targeted  system. 
These  references  should  contain  questionnaires  used  in  the  evaluations 
which  can  be  used  as  models  for  the  present  questionnaire.  Reviewing 
several  questionnaires  will  provide  the  system  evaluation  personnel  with 
an  ample  number  of  examples  from  which  to  choose  for  inclusion  in  their 
own  system  evaluation  questionnaire. 

2.  New  Question  Identification  -  Review  system  operation  and 
physical  description  references  for  material  not  addressed  in  similar 
system  evaluation  questionnaires.  Questions  should  address  functional 
grouping,  visibility,  anthropometric  reach  envelope  compatibility,  comfort, 
fatigue,  task  workload,  user  acceptance,  etc.  of  the  system  interface. 


Senior  Domain  Expert  Mapping 

Several  maps  are  created  representing  different  areas  of  the  SDE's 
expertise.  The  process  of  collecting  and  mapping  this  information  is  shown 
in  Figure  10  and  described  below. 

1.  SDE  Concept  Map  Acquisition  -  The  senior  domain  expert  is 
concept  mapped  to  elicit  all  the  system,  task  and  problem  data  known  to 
him/her.  This  includes  eliciting  all  the  knowledge  he/she  has  on  the  total 
operational  system,  the  scenarios  in  which  it  operates,  all  the 
tasks/functions  it  must  perform,  the  interface  design  and  the  problems 
associated  with  any  aspect  of  the  system.  Mapping  the  SDE  is  usually  an 
iterative  process.  A  top  level  map  may  first  be  taken  and  then  broken 
down  into  subcomponents  at  later  mapping  sessions. 
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Figure  9.  Questionnaire  Development  Process. 


2.  Task  Map  Acquisition  -  A  concept  map  of  all  personnel  tasks 
is  elicited  from  the  SDE  for  the  purpose  of  conducting  a  task  analysis  later. 
This  map  resembles  a  timeline  in  that  the  tasks  are  usually  sequential  in 
nature.  It  is  only  necessary  to  describe  the  tasks  globally  here  because 
more  detail  will  be  obtained  from  the  DEs. 

3.  Task  Map  Conversion  -  Specific  "nodes  of  inquiry"  can  be 
added  to  the  task  map  for  the  purpose  of  eliciting  specific  information 
from  domain  experts  on  certain  aspects  of  their  respective  tasks.  For  each 
task  node  there  are  a  pre-set  group  of  nodes  that  can  be  attached  to  it. 
These  include  nodes  representing  the  controls,  displays,  task  frequency, 
criticality  and  workload  ratings  and  known  problems  with  the  interface 
design  or  with  completing  the  task.  The  new  map  is  called  the  "task 
analysis  map." 

Collecting  data  for  task  workload,  criticality  and  frequency  and 
displays  and  controls  satisfaction  in  the  concept  map  format  combines  the 
best  of  two  worlds  for  a  comprehensive  evaluation  and  analysis.  The  DE 
system  task  analysis  map  provides  valuable  qualitative  data  regarding  the 
domain  expert's  relationship  with  the  system  along  with  the  quantitative 
data  necessary  for  conducting  traditional  task  analyses  and  workload 
assessments.  A  scaled  down  version  of  a  generic  "task  map"  is  shown  in 
Figure  11. 


4.  System  Map  Acquisition  -  A  concept  map  of  the  system 
hardware  is  gleaned  from  the  SDE  map  by  the  analysis  team  for  later 
comparison  with  system  hardware  diagrams  and  descriptions. 

5.  Problem  Map  Acquisition  -  A  concept  map  of  the  SDE's 
perceived  problems  with  the  system  is  gleaned  from  the  SDE  map.  This 
map  will  be  used  later  to  query  DEs  either  in  a  group  or  individually.  The 
map  will  be  used  to  solicit  further  details  on  the  SDE's  perceived  system 
problems. 
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Figure  11.  Scaled  Down  Example  of  the  Task  Map. 
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Domain  Expert  Mapping 


Individual  interviews  are  conducted  with  as  many  domain  experts  as 
possible.  In  these  interviews  a  system  interface  questionnaire  is 
administered,  and  a  task  map  and  a  problem  map  are  acquired.  This 
process  is  shown  in  Figure  12. 

1.  DE  System  Task  Map  Acquisition  -  The  section  of  the  DE  map 
describing  a  domain  expert's  main  tasks  will  have  already  been  outlined  in 
the  senior  domain  expert's  map.  The  top  level  tasks  provided  by  the  SDE 
will  provide  an  outline  for  a  more  detailed  map  which  all  DEs  will  expound 
upon.  The  DE's  are  asked  to  provide,  in  an  individual  interview,  specific 
details  concerning  the  following:  a  detailed  description  of  each  of  the 
subtasks  associated  with  the  main  tasks  outlined  by  the  SDE,  a  description 
of  how  he/she  performs  each  of  the  tasks/subtasks  outlined  by  the  senior 
domain  expert,  a  workload  rating  associated  with  each  task,  the  frequency 
of  each  task,  the  criticality  of  successful  completion  of  each  task,  the 
displays  and  controls  used  on  each  task,  how  and  why  they  are  used  in 
each  task,  a  satisfaction  rating  for  each  set  of  displays  and  controls,  and 
any  problems  associated  with  performing  one's  tasks  and  the  system 
interface  used  on  that  task.  Mapping  the  DE,  as  with  the  SDE,  is  an 
iterative  process  that  may  and  often  does  require  more  than  one  mapping 
session 


2.  DE  System  Problem  Map  Acquisition  -  A  group  of  DEs  is 
assembled  for  the  purpose  of  collecting  system  interface  problem  data  in 
concept  map  form.  Some  system  interface  problem  data  is  elicited  in  the 
process  of  constructing  the  DE  system  task  map.  However,  the  group 
mapping  process  has  proven  advantageous  in  that  the  problems  reported 
by  a  single  group  member  often  trigger  the  recall  of  other  problems 
encountered  by  other  group  members.  Some  of  these  problems  may  not 
be  recalled  in  the  individual  interview.  The  group  interview/mapping 
session  may  also  produce  interface  preference  data  and  possible  solutions 
to  reported  problems. 

3.  DE  Questionnaire  Administration  -  Questionnaires  are 
administered  to  as  many  DEs  as  possible.  The  questionnaires  address 
specific  system  interface  areas  of  interest  to  the  analysis  team.  These 
areas  may  be  ones  which  escape  the  attention  of  the  DEs  during  task  and 
problem  map  construction  or  ones  which  the  DEs  consider  to  be 
satisfactory  but,  in  fact,  are  good  sources  for  possible  improvement. 
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Figure  12.  Concept  Mapping  of  the  Domain  Expert. 
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Data/Product  Analysis 


The  data  analysis  step  of  the  system  evaluation  process  involves 
comparing  and  analyzing  the  products  of  the  previous  steps  (the  reference 
materials,  the  system  diagrams,  the  different  concept  maps,  the 
questionnaire  results,  etc.).  This  step  in  the  system  evaluation  process  is 
shown  in  Figure  13. 

In  making  the  following  comparisons  and  analyses  the  analysis  team 
looks  for  examples  of  poor  to  fair  interface  design  and/or  tasks 
characterized  by  medium  to  high  workload  levels.  Once  the  interface 
and/or  workload  problems  are  identified,  possible  solutions  are 
formulated.  The  formulated  solutions  are  subjected  to  formal  or  informal 
cost/benefit  analyses  to  determine  their  respective  feasibility. 

1.  System  hardware  vs.  SDE  System  Map  Comparison  -  This 
step  is  not  so  conscious  as  it  is  unconscious.  This  step  is  carried  out 
mentally  throughout  the  evaluation  process  where  the  analysis  team  is 
continually  making  mental  comparisons  between  the  SDE's  perception  of 
the  system  hardware  and  the  actual  system  hardware.  Differences 
between  the  two  can  be  rooted  in  the  interface  design.  The  differences 
need  to  be  identified  and  addressed  by  the  analysis  team. 

2.  Human  Factors  Guideline  Compliance  Check  -  The  system 
hardware  is  thoroughly  checked  to  make  sure  all  displays  and  controls 
conform  to  known  human  factors  interface  design  principles.  These 
principles  are  discussed  and/or  presented  in  several  different  references 
such  as  MIL-STD-46855,  MIL-STD-203,  MIL-STD-1333,  MIL-STD-1472D, 
AFSC  Design  Handbook  l-3;2-2,  The  Handbook  of  Perception  and  Human 
Performance  and  the  Human  Engineering  Guide  to  Equipment  Design. 

In  practice,  most  systems  are  not  in  total  compliance  with  all  of  the 
known  human  factors  guidelines.  The  reasons  for  this  are  usually 
mitigating  circumstances  particular  to  a  specific  system.  These 
circumstances  are  examined  to  verify  that  there  are  no  viable  alternatives 
which  would  bring  the  system  into  guideline  compliance.  The  objective  is 
not  to  bring  the  system  into  compliance  for  compliance's  sake  but  to  make 
the  system  as  user-friendly  as  possible  within  the  existing  monetary  and 
time  constraints.  An  accumulation  of  small  guideline  infractions  can 
steadily  increase  workload  to  the  point  of  causing  one  or  more  critical 
errors  due  to  operator  overload.  Therefore,  this  will  hopefully  prevent  an 
accumulation  of  minor  guideline  infractions  from  turning  into  a  major 
human  factors  system  problem. 
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3.  SDE  &  DE  Problem  Examination  -  This  step  is  characterized 
by  the  documentation  of  specific  problems  identified  by  the  SDE  and  DE's 
in  their  respective  system  and  problem  maps.  This  can  be  accomplished 
by  TAKE'S  "outline"  function  or  TAKE's  key-word  search  capability. 
Problems  reported  by  the  SDEs  and  DEs  can  either  be  identified  in  the  map 
outline  or  found  through  the  creation  of  "categories"  of  key-words  and  the 
automated  search  for  specific  information  related  to  those  "categories". 
Problems  related  to  the  "categories"  will  be  color-coded  in  the  map(s)  and 
therefore,  quite  easily  identified.  Possible  human  factors  solutions 
involving  system  interface  changes  or  task  automation  are  formulated  and 
discussed  by  the  analysis  team.  Several  factors  affecting  the 
determination  of  any  optimal  system  interface  or  automation  changes  are 
considered.  These  include  the  system  hardware,  the  task  frequency, 
workload  and  criticality  ratings,  the  "control  display  unit"  software 
architecture  or  design  and  the  overall  interface  design  or  layout. 

4.  Questionnaire  Analysis  -  The  questionnaires  are  analyzed  to 
determine  the  salient  opinions  of  the  system  interface  users  in  order  to 
pinpoint  serious  problems  with  the  interface  which  may  have  escaped  the 
attention  of  the  SDE's  or  DE's  in  the  generation  of  the  problem  and  system 
maps.  Statistical  analyses  can  be  applied  to  the  subjective  ratings  to 
determine  the  most  critical  or  serious  interface  subsystem  or  task 
problems  and/or  the  subsystems  or  tasks  which  are  rated  as  most 
satisfactory  by  the  users.  Once  the  most  critical  problem  areas  are 
determined  they  can  be  studied  for  possible  solutions.  Factors  affecting 
possible  solutions  to  problems  identified  in  the  questionnaire  are  listed 
above  in  the  section  describing  the  SDE  and  DE  problem  examination 
process. 


5.  Analysis  of  Task  Analysis  Map  -  First,  an  analysis  is 
conducted  on  the  workload  data  collected  in  each  DE  system  map  to 
determine  the  respective  workload  of  each  crew  member  s  tasks.  The 
mean  frequency  and  criticality  rating  of  each  task  can  also  be  computed. 
The  workload,  frequency  and  criticality  data  are  then  compared  to 
determine  which  tasks  or  task  combinations  are  good  candidates  for 
reassignment  to  other  crew  members  or  for  automation.  The  displays  and 
controls  used  for  each  high  workload  task  are  also  evaluated  for  possible 
human  factors  improvements.  Again,  the  TAKE  key-word  search  capability 
can  be  utilized  to  ferret  out  problematic  task  workload,  frequency  and 
criticality  data.  This  is  especially  helpful  when  dealing  with  very  large 
task  maps. 
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6.  Human  Factors  Guidelines  &  Problem  Map  Comparison  - 
Each  problem  in  the  problem  maps/list  is  examined  for  possible  violations 
of  documented  human  factors  interface  design  principles.  Solutions  to 
guideline  violations  are  formulated,  discussed  and  implemented  where 
possible.  Again,  the  aforementioned  factors  which  affect  the 
implementation  of  possible  solutions  "are  listed  in  the  section  above 
concerning  SDE  and  DE  problem  examination. 
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Figure  13.  Data  Analysis  Process. 


7.  Final  Product/System  Analysis  -  A  formal  or  informal 
cost/benefit  trade-off  study  can  be  conducted  to  determine  how  many  of 
the  identified  problems  can  be  solved.  The  relative  criticality  or  severity 
level  of  each  problem  is  estimated  by  the  analysis  team  based  on 
information  contained  in  the  SDE  and  DE  problem  maps.  The  severity  of 
each  problem,  the  time  and  money  costs  and  the  payoff  of  different 
possible  solutions  are  then  compared.  It  is  then  determined  at  which  point 
one  can  implement  interface  changes  as  solutions  to  the  identified 
problems.  This  analysis  is  often  conducted  in  an  informal  manner  through 
discussion  and  the  use  of  reference  materials. 
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CONCLUSIONS 


The  TAKE  methodology  has  proven  to  be  a  sound,  effective  process 
for  conducting  evaluations  of  human-system  interfaces.  Its  strengths  lie  in 
its  ability  to  a)  elicit  system-related  knowledge  or  information  from 
system  experts  or  users  and  b)  organize  that  information  in  a  manner 
which  is  conducive  to  logical  analysis.  The  TAKE  methodology  lends  itself 
to  different  problem  solving  approaches.  An  individual  users  approach  to 
mapping  the  source  of  the  system  information  should  be  based  on  the 
goals  of  the  evaluation.  Therefore,  the  focus  of  and  the  amount  or  type  of 
organization  imposed  upon  the  map  is  determined  by  both  the  mapper  and 
the  system  itself.  This  flexibility  gives  the  TAKE  tool  immense  power  and 
an  infinite  number  of  applications. 
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